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DETAILED ACTION 

1 . This Office Action is in response to the patent application number 09/997,704 
filed 11/30/2001. 

Information Disclosure Statement 

2. The Information Disclosure Statement submitted on 2/7/2003 is considered and 
entered into the file. 

Status of the claims 

3. The pending claims 1-80 are examined herein as follows. 

Claim Rejections - 35 USC § 1 02 
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that 
form the basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public 
use or on sale in this country, more than one year prior to the date of application for patent in the United 
States. 

4. Claims 1-5, 9-21, 25-37, 41-53. 57-69, and 73-80 are rejected under 35 
U.S.C. 102(b) as being anticipated by Hashimoto et al (US Pat No 5,632,002). 

The current invention relates to a speech interface for a graphical user interface 
(GUI) and a method for interfacing with a graphical user interface using the same. The 
current invention interacts (through speech) with the displayed screen objects ("screen 
areas"), Likewise, Hashimoto relates to a speech recognition interface system to be 
used as a man-machine interface in a data processing system such as personal 
computers. The speech interface enables audio input of words or vocabularies to 
transmit or communicate to applications. Similar to the current invention, Hashimoto 
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also interacts with the displayed screen objects ("screen areas"). These screen areas 
are nothing more that interface objects (text and/or graphical objects) displayed across 
the screen area. 

With regard to claims 1.17. 33, 49, and 65: 

As per "A method for providing speech control to a graphical user interface (GUI) 
containing objects to be selected," Hashimoto discloses a speech recognition interface 
method, system and computer readable medium containing instruction for enabling a 
speech control of a plurality of application programs executed on a computer (Abstract) 

As per "dividing the GUI into a plurality of screen areas;" as illustrated in several 
Figs of the drawing, such as for example Figs. 80A, 80B, etc, Hashimoto discloses a 
plurality of application program windows distributed across the screen area. 

As per "assigning priorities to the screen areas;" as illustrated, for example in 
Figs. 76-81, priorities are assigned (see table) to each application program windows 
distributed across the screen area shown in Figs. 80A and 80B (also see column 53, 
lines 1-40). 

As per "receiving a first audio input relating to the selection of one of the objects 
in the GUI;" Hashimoto also discloses that each application program window 
distributed across the screen area includes their own recognition vocabulary list 
(object), for example the focused SHELLTOOL application program window has 
selectable recognition vocabulary list (object) including "history," "process," " list," 
"home," and "finish" (see Fig. 29, etc). Again as illustrated in Fig. 80, the focused 
application window (SHELL TOOL) with a highest priority level will receive a first 
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speech input relating to the selection of one of the recognition vocabulary list (object) 
(column 53, lines 1-40). 

As per "determining the one of the screen areas that has the highest priority and 
includes a first object matching the first audio input;" as illustrated in Fig. 80A, for 
example, the focused SHELL TOOL application program window has the highest 
priority level so that it receives the first speech input (column 52, lines 1-23, column 53, 
lines 1-40) and the speech input is compared to recognition vocabulary list (object) of 
the SHELL TOOL, and if a match is found to the inputted speech, the matched 
recognized vocabulary is displayed in different form (e.g., highlighted) (see the 
recognized or matched speech vocabulary "copy", Fig, 21, or Fig. 53, #53F ( "mail 
today") (column 37, lines 44-54). 

As per "selecting the first object, if the determined screen area only contains one 
object matching the first audio input." Hashimoto also discloses comparing the speech 
input to the stored and recognized vocabulary list if the focused application program 
window contains the speech input vocabulary, selecting the first match vocabulary 
found from the determined application program window (column 19, lines 60-column 
20, lines 12, column 52, lines 1-23). 
With regard to claims 2, 18, 34, 50, and 66: 

As per "using a second input to select one of the objects that matches the first 
audio input in the determined screen area, if the determined screen area contains more 
than one object that matches the first audio input." Hashimoto discloses not only speech 
input but also a combination of keyboard and mouse inputs are also used to implement 
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his invention (see for example, column 22, lines 31-43). Hashimoto also discloses that if 
the speech input matches more than one vocabulary (object), then the recognized result 
(the matched object) can be transmitted to all the of the speech windows (application 
program windows) which share that same word (matched object), or to only the speech 
focused speech window (active window) at the time of the recognition (column 63, lines 
5-27). 

With regard to claims 3, 19, 35, 51, and 67: 

As per "marking each object matching the first audio input;" as illustrated in Fig. 
21 , Hashimoto also discloses shading or highlighted each word matching the speech 
input (see column 20, lines 43-50, Fig. 21, see marked "copy"). 

As per "receiving a second audio input relating to the selection of one of the 
marked objects" as illustrated in Fig. 53, mail too (53F) has received a speech focus 
(marked object). When the focused object is selected, it will display its own window as 
shown in Fig. 54, here a user further inputs speech such as "Supervisor" and 
"Emergency" in succession (column 37, lines 44-54, column 38, lines 32-40, Figs. 53- 
54). 

As per "selecting the marked object that best matches the second audio input." 
As illustrated in Figs-53-54, Hashimoto also discloses selecting the marked object (53F) 
that best matches the second audio input (column 37, lines 44-54, column 38, lines 32- 
40, Figs. 53-54). 

With regard to claims 4, 20. 36, 52, and 68: 
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As per "removing the markers." As illustrated in Figs. 80A-80B, the speech focus 
originally focused on the shell tool in FIG. 80A is changed (i.e. the marking is removed 
from shell tool) to that focused on the newly activated dtp system, such that the speech 
input for the dtp system becomes possible. 
With regard to claims 5. 21 . 37. 53. and 69: 

As per "marking the objects includes marking the objects with icons." Hashimoto 
discloses marking (e.g. shading) the object (e.g. copy, etc Fig. 21 ) with icon. Also as 
illustrated in Fig. 53, and elsewhere Hashimoto also discloses marking (shading, see 
Fig. 53) the objects (e.g., MAIL TOOL, 53F) with icon (i.e., mail tool icon). Similar 
illustrations are also shown in Fig. 69, and column 48, lines 10-14. 
With regard to claims 9. 25. 41 . 57, and 73: 

As per " assigning priorities to the screen areas includes assigning priorities to 
the screen areas based on usage." Hashimoto discloses wherein assigning priorities to 
the screen areas includes assigning priority level to each application program window or 
to each recognized vocabulary word that are distributed across the screen area based 
on usage, that is the focused application always assigned the highest priority unless an 
pop-up application window (e.g. emergency message) appears on the across screen. 
Furthermore, as illustrated in Figs. 76, 78, 81, etc, the application program windows are 
shown with their assigned priority level (column 42, lines 66-column 43, lines 8). 
With regard to claims 10. 26, 42. 58, and 74 

As per "assigning priorities to the screen areas includes assigning adjustable 
priorities to the screen areas." Hashimoto also discloses priority for the speech output 
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can be used effectively by setting or modifying it to a higher value in a case of 
emergency, such that the user's attention can be directed to it immediately, i.e., the 
task priority level setting (Fig. 81) for each application program window that are 
distributed across the screen area can be modified as needed specially during 
emergency (column 42, lines 66-column 43, lines 8). 
With regard to claims 1 1 . 27. 43. 59, and 75: 

As per "indicating the one of the screen areas having the highest priority." As 
illustrated in Figs. 76, 78, 81, 103, etc Hashimoto also discloses an application program 
with window having the highest priority level (see also column 53, lines 1-8). 
With regard to claims 12. 28, 44. 60. and 76: 

As per "indicating the highest priority screen area includes indicating the highest 
priority screen area with a visual output." As illustrated in Fig. 80A, the visual indicator 
(highlighted shaded lines) on shell tool application window indicates that the application 
has a focus and has the highest priority over the rest of the application, similarly, the dtp 
system in Fig. 80B, has a highest priority over the rest of the applications as shown with 
a visual (shaded lines) (see also column 53, lines 1-40). 
With regard to claims 13, 29, 45. 61. and 77: 

As per "indicating the highest priority screen area with visual output includes 
indicating the highest priority screen area with a visual output that includes 
highlighting." Hashimoto also discloses the visual output includes highlighting (see 
Figs. 21 , 28, 69, 80A-8B, etc.) Hashimoto further discloses that a window is focused 
(active) has a highest priority. The focus is indicated by displaying the window frame of 
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the focused window in thicker lines than the other windows, while the speech focus is 
indicated by changing the color of the title bar of the focused window from that of the 
other window (column 20, lines 43-50). 

With regard to claims 14, 30, 46, 62, and 78: 

As per "confirming the receipt of the first audio input." Hashimoto also discloses 
that in his vocabulary display program, whenever the speech recognition of the input 
speech is made, this vocabulary display program also receives the recognition result 
(confirmation) transmitted to the appropriate application program 5, such that the 
recognized word can be displayed in a color different from the rest as indicated by the 
shaded region in FIG. 21 , in order to facilitate the confirmation of the speech input 
recognized by the speech recognition system 3 (column 20, lines 43-50). 

With regard to claims 1 5, 31 , 47, 63, and 79: 

As per "confirming the receipt of the first audio input includes confirming the 
receipt of the first audio input with an audio output." Hashimoto discloses a speech 
recognition unit (SRU) 12 for receiving and recognizing an input speech and outputting 
a recognition result obtained therefrom; Hashimoto also discloses recognizing the 
receipt of the first speech (or audio input) includes recognizing the receipt of the first 
speech input with an speech output (Fig. 6, column 58, 1 1-27, column 71 , lines 14-21 , 
column 42-51). 

With regard to claims 16, 32, 48, 64, and 80: 

As per "A method for using a graphical user interface (GUI)," Hashimoto 
discloses a speech recognition interface method, system and computer readable 
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medium containing instruction for enabling a speech control of a plurality of application 
programs executed on a computer (Abstract). 

As per "receiving a first audio input relating to the selection of one of the objects 
in the GUI;" Hashimoto also discloses that each application program window 
distributed across the screen area includes their own recognition vocabulary list 
(object), for example the focused SHELLTOOL application program window has 
selectable recognition vocabulary list (object) including "history," "process," " list," 
"home," and "finish" (see Fig. 29). Again as illustrated in Fig. 80, the focused 
application window (SHELL TOOL) with a highest priority level will receive a first 
speech input relating to the selection of one of the recognition vocabulary list (object) 
(column 53, lines 1-40). 

As per "determining the one of the screen areas that has the highest priority and 
includes a first object matching the first audio input;" as illustrated in Fig. 80A, for 
example, the focused SHELL TOOL application program window has the highest 
priority level so that it receives the first speech input (column 52, lines 1-23, column 53, 
lines 1-40) and the speech input is compared to recognition vocabulary list (object) of 
the SHELL TOOL, and if a match is found to the inputted speech, the matched 
recognized vocabulary is displayed in different form (e.g., highlighted) (see the 
recognized or matched speech vocabulary "copy", Fig, 21, or Fig. 53, #53F ( "mail 
today") (column 37, lines 44-54). 

As per "selecting the first object, if the determined screen area only contains one 
object matching the first audio input;" Hashimoto also discloses comparing the speech 
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input to the stored and recognized vocabulary list if the focused application program 
window contains the speech input vocabulary, selecting the first match vocabulary 
found from the determined application program window (column 19, lines 60-column 20, 
lines 12, column 52, lines 1-23). 

As "using a second input to select one of the objects that matches' the first audio 
input in the determined screen area, if the determined screen area contains more than 
one object that matches the first audio input." Hashimoto discloses not only speech 
input but also a combination of keyboard and mouse inputs are also used to implement 
his invention (see for example, column 22, lines 31-43). Hashimoto also discloses that if 
the speech input matches more than vocabulary (object), then the recognized result (the 
matched object) can be transmitted to all the of the speech windows (application 
program windows) which share that same word (matched object), or to only the speech 
focused speech window at the time of the recognition (column 63, lines 5-27). 

Claim Rejections - 35 USC § 103 

The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

5. Claims 6-8, 22-24, 38-40, 54-56, and 70-72 are rejected under 35 U.S.C. 103(a) 

as being unpatentable over Hashimoto et al (5,632,002) in view of Tomonari Kamba et 

al. "Using small screen space more efficiently." April 13-18. 1996. CHI 96. 
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With regard to claims 6, 7, 22, 23, 38, 39. 54. 55. 70, and 71: 

Hashimoto discloses a recognized word (object) can be displayed in a color 
different from the rest as indicated by the shaded region (icon) in Figs. 21 , etc. Thus, 
Hashimoto discloses marking (e.g. shading) the object (e.g. copy, etc Fig. 21) with copy 
icon. Also as illustrated in Fig. 53, and elsewhere Hashimoto also discloses marking 
(shading, see Fig. 53) the objects (e.g., MAIL TOOL, 53F) with mail tool icon. 
Hashimoto also discloses overlapping or speech focused application window (e.g. Fig. 
28, etc). Hashimoto does not, however, suggest, or disclose that the shading or marking 
the objects with semi-transparenf icons that overlap the objects. 

Tomonari Kamba et al (Kamba) describes techniques for maximizing the 
efficient use of small screen space by combining delayed response with semi- 
transparency of control objects ("widgets") and on-screen text (abstract, Figs. 2 and 3). 
Kamba further discloses semi-transparent icons that overlap the objects. As shown 
along the bottom of the screen (Fig. 3) text objects are shown overlaying over several 
icons (also see page 384, second column, lines 1-11). 

Kamba and Hashimoto are analogous art because they are from the same field 
of endeavor, that is object interaction. 

At the time of the invention, it would have been obvious to a person of ordinary 
skill in the art to substitute the shaded icon of Hashimoto with semi-transparent widget 
(icon) of Kamba. 

The suggestion/motivation for doing so would have been to provide a more 
efficient use of screen space as suggested by Kamba (Kamba, abstract). 
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Therefore, it would have been obvious to combine Kamba and Hashimoto to obtain the 
invention as specified in claims 6.7, 22, 23, 38, 39, 54, 55, 70, and 71 . 
With regard to claims 8, 24, 40. 56, and 72: 

Hashimoto in view of Kamba further discloses marking the objects with 
semi-transparent icons that overlap the objects includes marking the objects with 
semi-transparent icons that have numbered labels. As illustrated in Fig. 103, each 
application window has a numbered label id. Also as illustrated in Fig. 116, each word 
also identified by its number label (see also Hashimoto, Figs. 58A, 58B, 103, 1 16, etc). 



6. Any inquiry concerning this communication or earlier communications from the 
Examiner should be directed to Tadesse Hailu, whose telephone number is (703) 306- 
2799. The Examiner can normally be reached on M-F from 10:00 - 6:30 ET. If attempts 
to reach the Examiner by telephone are unsuccessful, the Examiner's supervisor, John 
Cabeca, can be reached at (703) 308-31 1 6 Art Unit 21 73 CPK 2-4A51 . 

7. Any inquiry of a general nature or relating to the status of this application or 
proceeding should be directed to the Group receptionist whose telephone number is 
(703) 305-3900. 



Conclusion 



Tadesse Hailu 




August 17, 2004 



